Method and apparatus for robot path teaching

ABSTRACT

A dummy tool is used to teach a robot the path the robot will follow to perform work on a workpiece to eliminate the possibility of damaging an actual tool during the training. The dummy tool provides the robot programmer an indication of potential collisions between the tool and the workpiece and other objects in the work cell when path is being taught. The dummy tool can have a detachable input/output device with a graphic user interface (GUI) that can communicate wirelessly with the robot controller. The dummy tool can also have a moveable camera attached thereto to track the relationship of the tool to objects in the work area.

CROSS REFERENCE TO RELATED APPLICATIONS

This application claims the benefit of U.S. Provisional Application No.62/147,638, filed Apr. 15, 2015 and U.S. Provisional Application No.62/147,645, filed Apr. 15, 2015, which are incorporated herein byreference in their entirety.

TECHNICAL FIELD

The present application generally relates to teaching industrial robotsto follow a path so that the robot can perform work on a workpiece andmore particularly, but not exclusively, to using dummy tools, 3D visionand bare hands to train the robot.

BACKGROUND

An industrial robot can be automatically controlled, a multipurposemanipulator, and programmable in three or more axes. Industrial robotscan be located at a fixed position, independently mobile, or mounted ona device such as a motorized vehicle or a movable track or gantry.Hardware devices such as a teach pendant can be used by an operator to“jog” an industrial robot. “Jogging” teaches the robot points on a paththat the robot follows as the robot performs “work” on a workpiece.“Work” can be defined by actions performed by a robot such as painting,grinding, polishing, deburring, welding etc. that make a physical changeto the workpiece and/or other interactions that a robot has with aworkpiece, such as picking up and moving the workpiece from one locationto another. Some existing systems have various shortcomings relative tocertain applications. Accordingly, there remains a need for furthercontributions in this area of technology.

SUMMARY

One embodiment of the present invention is a unique system forprogramming a robot path. Other embodiments include apparatuses,systems, devices, hardware, methods, and combinations for teaching arobot to follow a desired path and perform work on a workpiece. Furtherembodiments, forms, features, aspects, benefits, and advantages of thepresent application shall become apparent from the description andfigures provided herewith.

BRIEF DESCRIPTION OF THE FIGURES

FIG. 1 is a block diagram for a robot system that is used to performwork on a workpiece according to one embodiment of the presentdisclosure;

FIGS. 2A-2C illustrate three examples of hand gestures to recordpositions in a work cell for robot path programming as well asoperations for the robot to perform during program execution accordingto one embodiment of the present disclosure;

FIGS. 2D-2E illustrates two examples of gestures for picking up anddropping a work object;

FIG. 3 is a flowchart illustrating one example of a method that useshand gestures to program a path that will be followed by an industrialrobot when work is performed on an object;

FIG. 4 depicts some exemplary forms of robot tools;

FIG. 5 shows examples of real milling tools and their correspondingdummy tool designs;

FIG. 6 shows an example of a two fingered gripper that can grip a partin alternate positions;

FIG. 7 shows a dummy tool with a detachable GUI; and

FIG. 8 shows a dummy tool with an attached camera.

DETAILED DESCRIPTION OF THE ILLUSTRATIVE EMBODIMENTS

For the purposes of promoting an understanding of the principles of theinvention, reference will now be made to the embodiments illustrated inthe drawings and specific language will be used to describe the same. Itwill nevertheless be understood that no limitation of the scope of theinvention is thereby intended. Any alterations and further modificationsin the described embodiments, and any further applications of theprinciples of the invention as described herein are contemplated aswould normally occur to one skilled in the art to which the inventionrelates.

Referring now to FIG. 1, there is shown a block diagram for a robotsystem 10 with an industrial robot 12 which is used to perform work on aworkpiece 14. The robot 12 is an automatic machine with ‘n’ number ofdegrees of freedom. System 10 can also include a vision sensor 11, acomputation device 13 and a robot controller 15. The vision sensor 11and computation device 13 can be in the same housing if sensor 11 is asmart sensor. The program to operate the industrial robot is in robotcontroller 15.

FIGS. 2A, 2B and 2C show examples of hand gestures that may be used torecord positions in a work cell for robot path programming as well asoperations for the robot to perform during program execution. Theseexemplary gestures can be used to teach a point on an object orworkpiece that the robot will perform work on and/or to follow a path toperform that work. The gesture illustrated in FIG. 2A can be used tostart the teaching, the gesture in FIG. 2B can be used to continue theteaching and the gesture in FIG. 2C can be used to stop the teaching.

In the exemplary embodiment, the start gesture and the stop gesture arethe same gesture; however, it should be appreciated that the robot canbe programmed to “understand” the context in which the two instructionsare used. For the start and stop instructions, the start instruction isgenerated first. Thus the first time that this hand gesture is used itwill be interpreted to mean the start instruction and the second timethe hand gesture is used it will be interpreted to mean the stopinstruction.

FIGS. 2D and 2E show two examples of gestures for picking up anddropping a work object respectively. These gestures can be used incombination with teaching the robot a point or to follow a path and thegestures can be taught by one hand or both hands.

Referring now to FIG. 3, there is shown a flowchart for a method 300that uses hand gestures to program a path to be followed by anindustrial robot when work is performed on an object. At step 302 theoperator positions the camera that is attached to the robot above thearea of interest. At step 304 a hand gesture is made by the operator topoint to a location in the robot scene. For example, if this location isthe start point the operator can use the gesture shown in FIG. 2A ifthat gesture has previously been associated with the start instructionto teach that robot target. As should be appreciated, if the robot hasto grab an object at that location, then another hand gesture such asthat shown in FIG. 2D can be used to teach that function.

At step 306, the image of the location pointing hand gesture of step 304and the associated location on the object are captured by the camera 11and sent to the computation device 13 for processing. The image capturedby the camera can be triggered by a defined hand gesture. At step 308,the computation device 13 calculates from the image the correspondinglocation and orientation of the robot tool in the robot scene.

While not shown in flowchart 300, after positioning the camera in step302, an optional step of acquiring an image of the robot scene withoutusing an operator's hand gesture can be performed in certainembodiments. If the image of the robot scene without the operator's handhas occurred in flowchart 300, then at step 308 the correspondinglocation and orientation of the robot tool in the robot scene can becalculated by subtracting the hand acquired in step 306. In anotherform, an optional 3D model of the operator's hand may be used at step308. The model can be used to identify the gesture and calculate thegesture information including geometry and other properties such asdeviation from a nominal position. By definition, the optional steps maynot be necessary to the image with the hand acquired in step 306 isprocessed so as to remove the hand without using an image that shows thehand or a 3D model of the hand.

In step 308, the hand and finger location and orientation can be used tocalculate the corresponding location and orientation of the robot toolin the robot scene. One way to calculate the robot target is tointersect the finger direction with the scene data. This can be done byhaving an image of the scene or a CAD created view of the scene. It isunderstood that both gesture image and scene image (or CAD) arereferenced in the same coordinate system and if not they can betransformed to a common coordinate system.

At step 310 the calculated location and orientation of the robot toolare sent to the computation device. Query 312 asks if more locationpoints are needed to complete the robot path. Query 312 can be definedby another gesture. If the answer is yes, the method 300 determines atquery 314 whether there is a need to reposition the camera. If theanswer to query 314 is no, then the method 300 returns to step 304 wherethe operator makes the hand gesture associated with the next locationpoint. If the answer to query 314 is yes, then the method 300 returns tostep 302 where the camera is repositioned. If the answer to query 312 isno, then method 300 ends as no more robot path points have to beacquired.

While not shown in FIG. 3, the function performed by step 310 can bereplaced with one of storage of the location and orientationinformation. If the answer to query 312 is that no more points have tobe taught, then the taught points can be sent to the robot. More detailson how 3D vision can be used with hand gestures for robot pathprogramming and the operations for the robot to perform during programexecution are described in published International ApplicationWO2014093822 and is incorporated herein by reference.

In some embodiments of the present disclosure, a “dummy tool” may beused to train the robot. Using a dummy tool to teach a pathway and/orwork operations for a workpiece will give the user or operator anindication of potential collisions between the tool and the work objectsand other objects in the work cell during the teaching or trainingwithout potentially causing damage to a real or actual tool during thepath programming. Feedback related to interference or potentialcollisions from using the dummy tool may or may not be 100% accuratedepending on how close the dimensions of the dummy tool are to that ofthe real or actual tool. Therefore after programming the robot with adummy tool a real simulation of the path may be performed with the realtool to determine if there will be any collisions between the real tooland the work object or other objects in the work cell during live oractual operation.

As shown in FIG. 4, robot tooling can come in various shapes and sizes.By way of example and not limitation, robot tooling can include amilling tool, a drilling tool, a grinding tool and a welding tool.Teaching path positions without an accurate representation of a tool'sshape and size can sometimes lead to teaching points that will causecollisions when the robot executes the program during live operation.However, using the actual tool to teach points may be undesirably formany reasons. For example the actual tool may be extremely heavy or mayhave dangerous features such as sharp edges or the like. A dummy toolwith substantially the same geometry as the real tool can be used toteach path positions using much of the same technology as the barehanded teaching techniques with an actual tool. FIG. 5 shows examples ofreal milling tools and their corresponding dummy tool designs. Adepiction of actual milling tools along with corresponding dummy toolsare shown to provide a visible example of potential differences.

In some instances a dummy tool can be connected to the robot duringcertain training operations and in other instances the dummy tool can beheld, moved, manipulated by an “operator” or “user” during certaintraining operations. The manipulation can include procedures to change aconfiguration state of the dummy tool such as opening or closing agripper and the like. The dummy tool can be used to teach and programthe robot with a 3D vision system. In some embodiments, one or morecameras associated with a 3D vision system can be used to track theposition, orientation and state of operating configuration of the dummytool during the training procedure. The 3D vision system can thentransmit the tracking data to a computational device so as to determinea desired operating pathway and other instructional commands for therobot to follow during actual operation.

Although not shown, other examples of the dummy tools may include agripper, spot welder or any other type of robot tooling. One desiredfeature of such dummy tools is that their shape and size substantiallymatches the real tool so that the path programmer has an accurate andintuitive understanding of possible collision issues and the tool'sproper orientation in the context of its use in the real work cell. Itis not necessary for the dummy tool used in path teaching to replicatethe real tool in its entirety. In some forms the dummy tool mayreplicate only part of the real tool especially those parts of the realtool that will be in contact with the object when the real tool performswork on the object. However, even if only a part of the dummy tool isused to teach the robot, the entire 3D model of the dummy tool may beused by the system to check for collision or calculate accurate robotpaths in simulation.

In some cases, there is one correct orientation of a tool which can bedetermined automatically from other geometric information readilyavailable, such as calculating the correct welding angle from the knownorientations of the surfaces being welded (e.g. the surfaces of a cornerweld). In other cases, there may be more than one valid orientation ofthe tool and in such cases the orientation of the dummy tool can be usedto determine the preferred orientation to generate the robot path.

FIG. 6 shows an example of a two fingered gripper 400 that can grip apart of workpiece 450 in more than one way. The gripper 400 may includefirst and second arms 402 and 403 that can be movable relative to oneanother in certain dummy tool configurations. The first and secondmovable arms 402 and 403 can include gripping surfaces 404 and 405 thatare configured to contact the workpiece 450 and perform tasks such asholding, manipulating or repositioning as required by the programmedrobot. The workpiece 450 can include a first set of opposing sides 452 aand 452 b that are engageable with the gripping arms surfaces 404 and405 respectively. In some forms the workpiece 450 may be engaged by thegripper 400 at a different orientation such as by opposing sides 454 aand 454 b of the workpiece 450. Using this tool an operator canintuitively, accurately and unambiguously teach the preferred grippingposition and orientation of the tool position relative to a workpiece450. In addition to the teaching a preferred gripping orientation, theuse of a dummy tool can also improve the accuracy of 3D vision basedpoint teaching because the dummy tool has a known geometry which the 3Dvision system can use for pattern matching and error correction in thevision data.

In some forms only the rough positioning of the dummy tool is necessary.This method can be the same as that of using only a bare hand technique.In this case, the position of the dummy tool is calculated from theinteraction between the gesture and scene model. In other forms an exactpositioning of the dummy tool is used to extract the robot path and anaccurate position of the dummy tool. In some forms a known 3D model ofthe dummy tool can be used. In such cases the dummy tool is defined witha specific configuration, then a pointer or a feature of interest on thedummy tool can be used along with the tool's 3D model to recognize,track and accurately determine the position of the tool. A dummy toolcan be used to teach positions and orientations of the robot. In someforms, hand gestures can be used along with the dummy tool to indicateoperations, such as turning a saw on or off, picking up a part, etc.

In some forms haptic feedback can be included with a system using adummy tool to indicate error states and/or other conditions of interest.An error state or invalid command can be defined by, for example, arequest to move the robot tool to an unreachable location, a prohibitedlocation, or an interference condition. For example, when the user triesto teach a point that is not actually reachable by the robot, a smallvibration and/or audible noise can be used to notify the user thatcommanded location is invalid. In other examples, haptic feedback can betransmitted to indicate that the dummy tool is near or crossing aboundary of certain zones (e.g. safety zones). Sometimes the hapticfeedback may be transmitted to a control panel or control device such asa joystick or the like. Other forms of haptic feedback may include anaudible sound to alert an operator that an issue relating to the dummytool needs to be addressed.

Referring now to FIG. 7, an input/output (I/O) device 500 can beremovably attachable to a dummy tool 400. The I/O device may include aGUI 501 (e.g. a graphical user interface such as a custom touch screeninput device). In some forms the I/O device 500 can include “smart”devices such as a mobile phone or a tablet. The I/O device 500 devicecan be attached to a dummy tool such as a gripper 400 to provideadditional functionality thereto. The I/O device 500 may be used toprovide command options such as the ability to record position, modifyposition, delete position, open gripper, close gripper, control I/Odisplay, etc. In some embodiments, the I/O device 500 may communicatewith the robot controller wirelessly and in other embodiments a directwire connection may be employed. In some forms the I/O device 500 may bededicated to a particular dummy tool and in other forms the I/O device500 may be attached and detached to and from any number of dummy tools.The user of the I/O device 500 can provide input commands including, byway of example and not limitation, a tap, swipe, or other interactionwith an icon shown in the GUI 501. The I/O device 500 may determinewhether a touch input has been detected and signal a confirmingresponse. As shown in FIG. 7 the I/O device 500 can have one or morebuttons 502 for dedicated commands. The I/O device 500 may be used toedit robot program data in response to touch interaction with standarduser interface elements such as with the buttons 502 or virtualkeyboards and the like. After inputting a command, the I/O device 500may transmit the updated robot program data to the robot controller.

Referring now to FIG. 8, an exemplary dummy tool 400 can include anattached camera 600 in certain embodiments. The camera 600 can be usedto track the relationship of the dummy tool 400 to objects in the workarea. In certain positions an object may be occluded from the camera'sview. To alleviate the occlusion, some forms of the dummy tool 400 andcamera 600 combination may include movable coupling means. In one aspectthe camera 600 may be manually repositioned by a user via attachmentmeans as would be known to those skilled in the art. In other forms thecamera 600 may include a motorized track 602 that permits electroniccontrol of the position of the camera with respect the dummy tool 400.In some forms the movement of the camera 600 is possible while the robotis in an operating mode such as moving along a pathway or performing (orsimulating) work on a workpiece. The camera 600 can be used to view orotherwise detect an object and determine the absolute location or therelative position of the dummy tool and the object in a definedcoordinate system. The position of the object may be used in the pathplanning. In some forms the camera 600 on the dummy tool 400 may also beused with the actual robot tool. In other forms the actual tool may nothave an attached camera or may have a different camera from that of thedummy tool 400. Furthermore, the camera 600 used with the dummy tooland/or the actual tool may be different from the camera used to trackthe gesture commands. For example, the tool camera could be a highresolution 2D camera that is capable of 2D pattern matching, while thecamera used to capture the hand gestures may be a 3D camera.

In one aspect, the present disclosure includes a system comprising: arobot configured to perform an operation on a workpiece; a 3D visionsystem operable for tracking objects; a dummy tool configured torepresent at least a portion of an actual tool used during operation ofthe robot; a database for storing robot commands defined by handgestures; a 3D model of a robot scene; and a computational deviceconfigured to define an operation path and operation commands for therobot to follow based on input from the training procedure includingdata captured by the 3D vision system of one or more positions andorientations of the dummy tool, hand gestures and the 3D model of arobot scene.

In refining aspects, the present disclosure includes a system whereinthe hand gestures are operable to provide instructions to the robot;wherein the operation on the workpiece includes at least one ofgrinding, milling, drilling, welding or maneuvering; comprising a cameraconnected to the dummy tool; wherein the camera is operable fortransmitting discrete pictures and/or a continuous video stream duringthe training procedure to the computational device; comprising acoupling constructed to permit movement of the camera with respect tothe dummy tool during operation of the robot; comprising a I/O deviceconnectable to the dummy tool; wherein the I/O device is operable fortransmitting commands or receiving data to/from the computationaldevice; wherein the commands to the I/O device include at least onerecording a position, modifying a position and changing a configurationstate of the dummy tool; comprising a haptic feedback mechanism operablefor generating haptic feedback when the robot encounters an obstacle orreceives an invalid command during the training procedure.

In another aspect, the present disclosure includes a robot systemcomprising: a programmable robot; a sensor for sensing input commandsfrom a hand gesture; a 3D camera operable to provide a 3D robot scene; adummy tool configured to replicate at least a portion of an actual tool;wherein a pathway and operational movement of the robot is defined byinstructions from a computational device based on results from atraining procedure that uses the robot and the dummy tool.

In refining aspects, the present disclosure includes a robot systemcomprising means for providing haptic feedback to an operator upon anoccurrence of predefined criteria; wherein the predefined criteriaincludes one of an error state or invalid command; wherein the hapticfeedback includes one of a vibratory output and an audible sound;further comprising a camera positioned on the dummy tool; wherein thecamera is a 2D camera; wherein the camera is connected to the dummy toolvia a movable coupling; wherein the camera is movable with respect tothe dummy tool while the robot is moving and/or performing operations onthe workpiece; further comprising a detachable I/O device connected tothe dummy tool; wherein an operator holds and moves the dummy tool byhand during at least a portion of the training procedure; wherein therobot holds and moves the dummy tool during at least a portion of thetraining procedure.

In another aspect, the present disclosure includes a method comprising:providing a dummy tool for operation with a robot; training the robot tomove along a workpath and/or work on a workpiece using the dummy tool todetermine a potential interference with an object along the workpath;and inputting commands to a computational device to define the workpathand/or operational maneuvering of the robot with an actual tool based onthe training.

In refining aspects, the present disclosure includes a method whereinthe training further comprises using hand gestures for command input;wherein the training further comprises transmitting haptic feedback whenthe robot encounters and error state or an invalid command; wherein thetraining further comprises attaching a camera to the dummy tool andtransmitting video images to the computational device; wherein theattaching includes a movable coupling operable for attaching the camerato the dummy tool such that the camera is movable relative to the dummytool when the robot is working on a workpiece; further comprisingteaching the robot to engage a workpiece in a preferred location withthe dummy tool; wherein the training further comprises holding, moving,manipulating and changing a configuration state of the dummy tool with ahand of an operator; wherein the training further comprises holding,moving, manipulating and changing a configuration state of the dummytool with the robot.

While the invention has been illustrated and described in detail in thedrawings and foregoing description, the same is to be considered asillustrative and not restrictive in character, it being understood thatonly the preferred embodiments have been shown and described and thatall changes and modifications that come within the spirit of theinventions are desired to be protected. It should be understood thatwhile the use of words such as preferable, preferably, preferred or morepreferred utilized in the description above indicate that the feature sodescribed may be more desirable, it nonetheless may not be necessary andembodiments lacking the same may be contemplated as within the scope ofthe invention, the scope being defined by the claims that follow. Inreading the claims, it is intended that when words such as “a,” “an,”“at least one,” or “at least one portion” are used there is no intentionto limit the claim to only one item unless specifically stated to thecontrary in the claim. When the language “at least a portion” and/or “aportion” is used the item can include a portion and/or the entire itemunless specifically stated to the contrary.

Unless specified or limited otherwise, the terms “mounted,” “connected,”“supported,” and “coupled” and variations thereof are used broadly andencompass both direct and indirect mountings, connections, supports, andcouplings. Further, “connected” and “coupled” are not restricted tophysical or mechanical connections or couplings.

What is claimed is:
 1. A system comprising: a 3D vision system operablefor tracking objects; a dummy tool configured to represent at least aportion of an actual tool used during operation of a robot; a 3D modelof a robot scene; and a computational device configured to define anoperation path and operation commands for the robot to follow based oninput from a training procedure including data captured by the 3D visionsystem of one or more positions and orientations of the dummy tool andthe 3D model of a robot scene.
 2. The system of claim 1, which furtherincludes a database for storing robot commands defined by hand gestures,and wherein the computational device is further configured to define theoperation path and operation commands for a robot to follow based onhand gestures.
 3. The system of claim 2, which further includes therobot configured to perform an operation on a workpiece.
 4. The systemof claim 3, wherein the hand gestures are operable to provideinstructions to the robot.
 5. The system of claim 3, wherein the robotis a mobile robot, and which further includes an I/O device connectableto the dummy tool, and wherein the I/O device is operable fortransmitting commands or receiving data to/from the computationaldevice.
 6. The system of claim 5, wherein the commands to the I/O deviceinclude at least one recording a position, modifying a position andchanging a configuration state of the dummy tool.
 7. The system of claim2, wherein the operation on the workpiece includes at least one ofgrinding, milling, drilling, welding or maneuvering.
 8. The system ofclaim 2, further comprising a haptic feedback mechanism operable forgenerating haptic feedback when the robot encounters an obstacle orreceives an invalid command during the training procedure.
 9. The systemof claim 8, wherein the haptic feedback includes one of a vibratoryoutput and an audible sound.
 10. The system of claim 8, wherein thehaptic feedback mechanism is activated to indicate an error state.
 11. Asystem comprising: a 3D camera operable to provide a 3D robot scene; adummy tool configured to replicate at least a portion of an actual tool;wherein a pathway and operational movement of a robot is defined byinstructions from a computational device based on results from atraining procedure that uses data from the 3D camera related to thedummy tool.
 12. The system of claim 11, which further includes a sensorfor sensing input commands from a hand gesture.
 13. The system of claim12, which further includes the robot and a database for storing robotcommands defined by hand gestures.
 14. The system of claim 13, whereinthe robot is mobile, and wherein the database includes a query commanddefined by a query hand gesture, the query command structured todetermine whether additional location points are required to completethe pathway and operational movement of the robot.
 15. The system ofclaim 11, further comprising means for providing haptic feedback to anoperator upon an occurrence of predefined criteria.
 16. The system ofclaim 15, wherein the predefined criteria includes one of an error stateor invalid command.
 17. The system of claim 16, wherein an operatorholds and moves the dummy tool by hand during at least a portion of thetraining procedure, and wherein the haptic feedback includes one of avibratory output and an audible sound.
 18. The system of claim 11,further comprising a camera connected to the dummy tool, wherein thecamera is operable for transmitting discrete pictures and/or acontinuous video stream during the training procedure to thecomputational device.
 19. The system of claim 18, further comprising acoupling constructed to permit movement of the camera with respect tothe dummy tool during operation of the robot.
 20. The system of claim19, which further includes a sensor for sensing input commands from ahand gesture, which further includes a programmable robot, and whichfurther includes a database for storing robot commands defined by handgestures, and wherein the robot is mobile.
 21. A method comprising:manipulating a dummy tool structured to mimic an actual tool; capturingan image of the dummy tool using a 3D vision system; training a robotsystem to form robot commands useful to move a robot along a workpathand/or work on a workpiece using the dummy tool to determine a potentialinterference with an object along the workpath.
 22. The method of claim21, wherein the training includes utilizing a 3D vision system toprovide the robot with a vision scene which it uses to detect the dummytool.
 23. The method of claim 22, which further includes inputtingcommands to a computational device to define the workpath and/oroperational maneuvering of the robot with the actual tool based on thetraining.
 24. The method of claim 22, wherein the training furthercomprises using hand gestures for command input.
 25. The method of claim24, wherein the training further comprises transmitting haptic feedbackwhen the robot encounters and error state or an invalid command.
 26. Themethod of claim 25, wherein the training further comprises attaching acamera to the dummy tool and transmitting video images to thecomputational device.
 27. The method of claim 24, which further includescapturing data related to an operation configuration of the dummy tool.28. The method of claim 24, which further includes simulating a robotpath defined as a result of the training to check for collision orcalculate accurate robot paths.